{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d09fc8a0-9024-49a6-91e8-1aa851535ea1",
   "metadata": {},
   "source": [
    "# NeMo Curator Download and Extract Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4913b883-3b62-49c3-b08f-426c9dc43047",
   "metadata": {},
   "source": [
    "## About This Notebook\n",
    "NeMo Curator has pre-built download and extract pipelines for **Common Crawl**, **Wikipedia** and **ArXiv** datasets. In this tutorial, we will introduce how to execute these pipelines.\n",
    "\n",
    "For more information about downloading and extracting data with NeMo Curator, refer to the [Download Data](https://docs.nvidia.com/nemo/curator/latest/curate-text/load-data/index.html) and [Data Acquisition Concepts](https://docs.nvidia.com/nemo/curator/latest/about/concepts/text/data-acquisition-concepts.html) documentation pages."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c90fe3d",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "`wget` will be needed in the tutorial for data downloading."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d06f3ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "!apt update\n",
    "!apt install -y wget"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6c131c1",
   "metadata": {},
   "source": [
    "To run a pipeline in NeMo Curator, we must start a Ray cluster. This can be done manually (see the [Ray documentation](https://docs.ray.io/en/latest/ray-core/starting-ray.html)) or with Curator's `RayClient`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1570ed60",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nemo_curator.core.client import RayClient\n",
    "\n",
    "try:\n",
    "    ray_client = RayClient()\n",
    "    ray_client.start()\n",
    "except Exception as e:\n",
    "    msg = f\"Error initializing Ray client: {e}\"\n",
    "    raise RuntimeError(msg) from e"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5967b66a-94ba-4392-a1bf-58ae3b3add9e",
   "metadata": {},
   "source": [
    "## Common Crawl Download and Extract\n",
    "\n",
    "### About Common Crawl Dataset\n",
    "The [Common Crawl](https://commoncrawl.org/) dataset is a massive, openly available archive of web data collected by the nonprofit Common Crawl organization since 2008. It consists of petabytes of raw web page data, metadata extracts, and text extracts, providing a valuable resource for anyone interested in large-scale web analysis or AI research. Common Crawl currently stores the crawl data using the [Web ARChive (WARC)](https://en.wikipedia.org/wiki/WARC_(file_format)) format. In this section, we will explore how to use NeMo Curator to download the Common Crawl raw data (WARC files) and extract the text from the it.\n",
    "\n",
    "More information about downloading Common Crawl data with NeMo Curator can be found in the [documentation](https://docs.nvidia.com/nemo/curator/latest/curate-text/load-data/common-crawl.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1f5ecef-39f1-44b8-97b4-76b82a282c71",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nemo_curator.pipeline.pipeline import Pipeline\n",
    "from nemo_curator.stages.text.download.common_crawl.stage import CommonCrawlDownloadExtractStage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "904ad6d4-a531-43c8-949a-45dc7b762e80",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p data/common_crawl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04d56978-e357-43a4-8c4b-6677329c168f",
   "metadata": {},
   "outputs": [],
   "source": [
    "stage = CommonCrawlDownloadExtractStage(\n",
    "    start_snapshot=\"2025-30\",\n",
    "    end_snapshot=\"2025-30\",\n",
    "    download_dir=\"data/common_crawl\",\n",
    "    crawl_type=\"main\",\n",
    "    url_limit=2,\n",
    "    record_limit=1000,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b552dbb-31a7-4297-a4da-48ebd5fa78c1",
   "metadata": {},
   "source": [
    "`CommonCrawlDownloadExtractStage` is the pre-built download and extract pipeline for the Common Crawl dataset, including 4 core components:\n",
    "1. URL Generator: gets the URLs for Common Crawl data\n",
    "2. Downloader: downloads the WARC files from the Common Crawl to a local directory\n",
    "3. Iterator: extracts contents from the downloaded WARC files\n",
    "4. Extractor: extracts text from HTML content\n",
    "\n",
    "Core arguments for this class:\n",
    "* `start_snapshot`: the first Common Crawl snapshot that will be included in the download. For CC-MAIN datasets, use the format 'YYYY-WeekNumber' (e.g., '2020-50' or '2021-04'); for CC-NEWS datasets (when `crawl_type=\"news\"`), use the 'YYYY-MM' (Year-Month) format. A list of valid Common Crawl snapshots can be found [here](https://data.commoncrawl.org/).\n",
    "* `end_snapshot`: the last Common Crawl snapshot that will be included in the download, it should be the same format as `start_snapshot`.\n",
    "* `download_dir`: the location of the downloaded snapshots will be stored.\n",
    "* `crawl_type`: optional; the type of Common Crawl dataset, it could be `\"main\"` (default) or `\"news\"`.\n",
    "* `url_limit`: optional; the maximum number of WARC files to download from the snapshot range.\n",
    "* `record_limit`: optional; the maximum number of records to extract from each WARC file.\n",
    "\n",
    "As the tutorial is tested in a single node machine with limited resource, we set the `url_limit` and `record_limit` to a small number for demonstration purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bedcd5b-eefe-4ef4-920b-7158e68f2c4a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pipeline = Pipeline(\"common_crawl\")\n",
    "pipeline.add_stage(stage)\n",
    "results = pipeline.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81168599-a76c-41b4-95ef-ab1adcdcb05d",
   "metadata": {},
   "source": [
    "After running above code, the Common Crawl `warc.gz` files should be downloaded to the `download_dir`.\n",
    "\n",
    "The extracted texts are stored in memory in the `results` variable, which is a list of `nemo_curator.tasks.document.DocumentBatch` objects.\n",
    "The length of `results` is the same as the `url_limit` (i.e., the number of WARC files downloaded). \n",
    "\n",
    "`results[i].data` is a Pandas Dataframe holding the extracted texts and related metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "487bf6f2-a5d7-4208-a452-6c8c7679fc5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"len(results): {len(results)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e48414a7-03ae-42be-a72e-c7876e8bdffa",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "results[0].data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b194ee93-d1d6-49ab-8453-52e3a81f5ff7",
   "metadata": {},
   "source": [
    "We can use `JsonlWriter` to save the Pandas Dataframe as `jsonl` files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "214b5673-cd6e-446c-9527-8275c22701b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p out_jsonl/common_crawl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd3a18ea-d225-44d5-ad71-1d53c6ece204",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nemo_curator.stages.text.io.writer.jsonl import JsonlWriter\n",
    "\n",
    "output_dir = \"out_jsonl/common_crawl\"\n",
    "jsonl_writer = JsonlWriter(path=output_dir, write_kwargs={\"force_ascii\": False})\n",
    "\n",
    "for idx, result in enumerate(results):\n",
    "    jsonl_writer.write_data(result, f\"{output_dir}/{idx}.jsonl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d07b2646-3044-42da-a217-98c9780cd2cd",
   "metadata": {},
   "source": [
    "At this point, the `jsonl` files should be located in `out_jsonl/common_crawl`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21d1726d-b490-46d9-ae25-7585b674cda5",
   "metadata": {},
   "source": [
    "## Wikipedia Download and Extract\n",
    "\n",
    "### About Wikipedia Dataset\n",
    "The [Wikipedia](https://dumps.wikimedia.org/backup-index.html) dataset contains a complete copy of all Wikimedia wikis, in the form of wikitext source and metadata embedded in XML. These snapshots are provided at the very least monthly and usually twice a month. In this section, we will introduce how to download the Wikipedia dump files (`.bz2`) and extract the text from these raw files using NeMo Curator.\n",
    "\n",
    "More information about downloading Wikipedia data with NeMo Curator can be found in the [documentation](https://docs.nvidia.com/nemo/curator/latest/curate-text/load-data/wikipedia.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f45ff95-6374-4971-9f73-12676b6592d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nemo_curator.pipeline.pipeline import Pipeline\n",
    "from nemo_curator.stages.text.download.wikipedia.stage import WikipediaDownloadExtractStage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d085fb4-2baa-4174-8749-4c3a9f3d4ee8",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p data/wikipedia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05a44629-0a92-4ff4-a8ae-35a05ad88a75",
   "metadata": {},
   "outputs": [],
   "source": [
    "stage = WikipediaDownloadExtractStage(language=\"fr\", download_dir=\"data/wikipedia\", url_limit=2, record_limit=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5500416-50a5-4ee9-bd02-2f81537a7043",
   "metadata": {},
   "source": [
    "Similar to `CommonCrawlDownloadExtractStage`, `WikipediaDownloadExtractStage` is the pre-built download and extract pipeline for the Wikipedia dataset, including 4 core components:\n",
    "1. URL Generator: gets the URLs for Wikipedia dump files\n",
    "2. Downloader: downloads the Wikipedia dump files (`.bz2`) from wikimedia.org to a local directory\n",
    "3. Iterator: processes downloaded Wikipedia dump files and extracts article content\n",
    "4. Extractor: extracts for Wikipedia articles from MediaWiki XML dumps\n",
    "\n",
    "Core arguments for this class:\n",
    "* `language`: Language code for the Wikipedia dump (e.g., \"en\", \"es\", \"fr\"), a full list of language code could be found [here](https://www.wikidata.org/wiki/Help:Wikimedia_language_codes/lists/all)\n",
    "* `download_dir`: Directory to store downloaded `.bz2` files\n",
    "* `dump_date`: Specific dump date in \"YYYYMMDD\" format (if None, uses latest)\n",
    "* `wikidumps_index_prefix`: Base URL for Wikipedia dumps\n",
    "* `verbose`: If True, enables verbose logging\n",
    "* `url_limit`: Maximum number of dump URLs to process\n",
    "* `record_limit`: Maximum number of articles to extract per file\n",
    "* `add_filename_column`: Whether to add filename column to output\n",
    "* `log_frequency`: How often to log progress during iteration\n",
    "\n",
    "A list of latest Wikipedia dump files could be found [here](https://dumps.wikimedia.org/backup-index-bydb.html).\n",
    "\n",
    "As the tutorial is tested in a single node machine with limited resource, we set the `url_limit` and `record_limit` to a small number for demonstration purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bc2349b-fb08-4858-b2cb-f2d4d3821838",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pipeline = Pipeline(\"wikipedia\")\n",
    "pipeline.add_stage(stage)\n",
    "results = pipeline.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e73a2640-3d1d-4be2-91f6-d3176f6d8456",
   "metadata": {},
   "source": [
    "After running above code, the Wikipedia `.bz2` dump files should be downloaded to the `download_dir`.\n",
    "\n",
    "`results[i].data` is a Pandas Dataframe holding the extracted texts and related metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc4432be-7edb-4dcc-a679-3ca58e34cb4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"len(results): {len(results)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74406cb3-29d5-4650-a251-599d6a2836d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "results[0].data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1459218-3a94-4a73-8149-d9144ada348e",
   "metadata": {},
   "source": [
    "We can use `JsonlWriter` to save the Pandas Dataframe as `jsonl` files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cad96f0-a531-47b0-8341-c9997e1bac9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir out_jsonl/wikipedia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86fd0e72-4fa8-495c-a467-cba02d516524",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nemo_curator.stages.text.io.writer.jsonl import JsonlWriter\n",
    "\n",
    "output_dir = \"out_jsonl/wikipedia\"\n",
    "jsonl_writer = JsonlWriter(path=output_dir, write_kwargs={\"force_ascii\": False})\n",
    "\n",
    "for idx, result in enumerate(results):\n",
    "    jsonl_writer.write_data(result, f\"{output_dir}/{idx}.jsonl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a86085c9-9554-4535-a1b3-e2ba1a31035b",
   "metadata": {},
   "source": [
    "At this point, the `jsonl` files should be located in `out_jsonl/wikipedia`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2a7774e-6d46-4855-90eb-f79e455bfc06",
   "metadata": {},
   "source": [
    "## ArXiv Download and Extract\n",
    "\n",
    "### About ArXiv Dataset\n",
    "[ArXiv](https://info.arxiv.org/help/bulk_data_s3.html) is an open access research sharing platform and access to bulk data is also open, with certain stipulations. The ArXiv source files (mostly TeX/LaTeX with figures in `tar.gz` format) are available from Amazon S3 in requester pays buckets, and the downloader should pay Amazon for the download based on bandwidth used. In this section, we will walk through how to leverage NeMo Curator to download and extract the ArXiv dataset.\n",
    "\n",
    "More information about downloading ArXiv data with NeMo Curator can be found in the [documentation](https://docs.nvidia.com/nemo/curator/latest/curate-text/load-data/arxiv.html).\n",
    "\n",
    "### Prerequisites\n",
    "Before proceeding, we need to set **AWS credentials** in the environment as the data are stored in AWS S3 [requester pays buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html) (pricing details could be found [here](https://aws.amazon.com/s3/pricing/))."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef101752-aa99-470a-b587-4548b83c1625",
   "metadata": {},
   "source": [
    "#### Set your AWS Credentials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "193812d1-69b0-46d7-a1f4-525e52573671",
   "metadata": {},
   "outputs": [],
   "source": [
    "%env AWS_ACCESS_KEY_ID=\n",
    "%env AWS_SECRET_ACCESS_KEY=\n",
    "%env AWS_SESSION_TOKEN="
   ]
  },
  {
   "cell_type": "markdown",
   "id": "988105cf-5b4e-4288-be1d-4c4219c98bf9",
   "metadata": {},
   "source": [
    "#### Ensure the ArXiv S3 bucket accessibility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e01ee93b-e8f3-4455-a4b6-ae4fae41e3b5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!s5cmd --request-payer=requester ls s3://arxiv/src/ | head | grep '.tar'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c914c965-6670-4eb6-9e37-2c2653f297c6",
   "metadata": {},
   "source": [
    "If everything is set, the above command should be able to list the `.tar` files in the S3 bucket like:\n",
    "```\n",
    "2010/12/23 05:14:01         227420160  arXiv_src_0001_001.tar\n",
    "2010/12/23 05:18:10         228853760  arXiv_src_0002_001.tar\n",
    "2010/12/23 05:22:17         232980480  arXiv_src_0003_001.tar\n",
    "2010/12/23 05:26:33         193167360  arXiv_src_0004_001.tar\n",
    "2010/12/23 05:30:12         257617920  arXiv_src_0005_001.tar\n",
    "2010/12/23 05:34:58         244418560  arXiv_src_0006_001.tar\n",
    "2010/12/23 05:39:30         247439360  arXiv_src_0007_001.tar\n",
    "2010/12/23 05:44:19         289003520  arXiv_src_0008_001.tar\n",
    "2010/12/23 05:49:21         232693760  arXiv_src_0009_001.tar\n",
    "2010/12/23 05:53:40         280913920  arXiv_src_0010_001.tar\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3b4fdd8-4ef6-4b73-931f-1009922c2718",
   "metadata": {},
   "source": [
    "### Run ArXiv Download and Extract Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13d1861d-8d8c-41ba-bed6-3fa5367f5dda",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nemo_curator.pipeline.pipeline import Pipeline\n",
    "from nemo_curator.stages.text.download.arxiv.stage import ArxivDownloadExtractStage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0a5c01f-930d-46eb-9448-9b277c832041",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p data/arxiv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3530f8a-df28-45c7-bde3-f44e4386dddb",
   "metadata": {},
   "outputs": [],
   "source": [
    "stage = ArxivDownloadExtractStage(download_dir=\"data/arxiv\", url_limit=2, record_limit=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ac39cda-ab02-47ef-b69f-705f9fc33ebe",
   "metadata": {},
   "source": [
    "Similar to previous examples, the `ArxivDownloadExtractStage` pipeline obtains a list of ArXiv tar file URLs, downloads the tar files, and then extracts the contained LaTeX source files. \n",
    "\n",
    "Core arguments for this class:\n",
    "* `download_dir` (str, optional): The directory where the raw downloaded tar files will be kept. Defaults to \"./arxiv_downloads\".\n",
    "* `url_limit` (Optional[int], optional): Limits the maximum number of ArXiv tar file URLs to download and process. If None, all available URLs (from get_arxiv_urls) are processed.\n",
    "* `record_limit` (Optional[int], optional): Limits the maximum number of records to extract from each tar file. If None, all available records are extracted.\n",
    "* `add_filename_column` (bool | str, optional): If True, adds a column to the output DataFrame with the filename of the tar file. If a string, adds a column with the specified name. Defaults to True.\n",
    "* `log_frequency` (int, optional): How often to log progress. Defaults to 1000.\n",
    "* `verbose` (bool, optional): If True, prints verbose output. Defaults to False.\n",
    "\n",
    "As the tutorial is tested in a single node machine with limited resource, we set the `url_limit` and `record_limit` to a small number for demonstration purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7802061f-0f7a-4515-acbe-3a7ea6737f4d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pipeline = Pipeline(\"arxiv\")\n",
    "pipeline.add_stage(stage)\n",
    "results = pipeline.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd6838a0-7f88-4b09-99ad-49d8dfa61c0e",
   "metadata": {},
   "source": [
    "After running above code, the ArXiv `.tar` files should be downloaded to the `download_dir`.\n",
    "\n",
    "`results[i].data` is a Pandas Dataframe holding the extracted texts and related metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebc90ba8-fea8-4a45-bb89-8169dea4f82b",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"len(results): {len(results)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a2a4301-af9c-4312-9900-24d7e6e1d010",
   "metadata": {},
   "outputs": [],
   "source": [
    "results[0].data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "656966ca-4837-40ad-a74a-f41e00c0eab6",
   "metadata": {},
   "source": [
    "We can use `JsonlWriter` to save the Pandas Dataframe as `jsonl` files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "686cb075-f227-4345-8c82-a5913a3ec85c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir out_jsonl/arxiv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b44a7c4c-231a-4440-a501-9895f3704c38",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nemo_curator.stages.text.io.writer.jsonl import JsonlWriter\n",
    "\n",
    "output_dir = \"out_jsonl/arxiv\"\n",
    "jsonl_writer = JsonlWriter(path=output_dir, write_kwargs={\"force_ascii\": False})\n",
    "\n",
    "for idx, result in enumerate(results):\n",
    "    jsonl_writer.write_data(result, f\"{output_dir}/{idx}.jsonl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31d9a0c9-ebbf-4efa-8ce5-991530b4a91f",
   "metadata": {},
   "source": [
    "At this point, the `jsonl` files should be located in `out_jsonl/arxiv`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b692cc64",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Since the pipelines ran to completion and the results were written to JSONL files, we can shut down the Ray cluster with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e9c2f7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    ray_client.stop()\n",
    "except Exception as e:  # noqa: BLE001\n",
    "    print(f\"Error stopping Ray client: {e}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7101e4e1",
   "metadata": {},
   "source": [
    "In this tutorial, we have introduced how to download and extract the Common Crawl, Wikipedia and ArXiv datasets with NeMo Curator. Apart from the built-in pipelines, developers can also create a custom download and extract pipeline for other data sources. The framework follows a 4-step pipeline pattern where each step is implemented as an abstract base class with corresponding stages:\n",
    "\n",
    "```\n",
    "1. URLGenerator → URLGenerationStage    (URLs from config/input)\n",
    "2. DocumentDownloader → DocumentDownloadStage    (local files from URLs)\n",
    "3. DocumentIterator → DocumentIterateStage    (raw records from files)\n",
    "4. DocumentExtractor → DocumentExtractStage    (structured data from records)\n",
    "```\n",
    "\n",
    "For more details, you can refer to the source code of these pipelines and the README file in `nemo_curator/stages/text/download/`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
